Smart cable for redundant tor&#39;s

ABSTRACT

In a rack comprising a group of servers and at least two top-of-rack switches, a link fault is detected. A smart data cable connects each of the servers to both top-of-rack switches. A control signal indicates an active communication path from one of the top-of-rack switches to the servers. In response to detecting a failure of the active communication path, the control signal indicates a switch to the second of the top-of-rack switches. In response to the updated control signal, a switching mechanism of the data cable changes the active communication path to the second of the top-of-rack switches.

BACKGROUND

A data center is a facility that houses computer systems and variousnetworking, storage, and other related components. Data centers may, forexample, provide computing services to businesses and individuals as aremote computing service or provide “software as a service” (e.g., cloudcomputing).

A data center may house hundreds or thousands of servers. Each servermay host a number of virtual machines and other resources. It isimportant to prevent downtime due to hardware and network failures andother issues that may prevent the operation of services provided by thedata center. Some data centers may implement ways to provide someresiliency to failures that might prevent a loss of networkcommunications. Such resiliency may exist from Tier-1 networkingelements to the higher tier networking elements. However, since eachindividual server typically only uses a single connection to the firstnetwork element, referred to as a Tier-0 element (e.g., Top of Rack(ToR) device), there may be a single point of failure that may isolate aserver or an entire rack of servers from the network.

When a data center experiences server connectivity issues, loss of dataand services may result, preventing users from providing qualityservices to their downstream customers, which may result in lost revenueand customer dissatisfaction. Production loss and inefficiencies withrespect to computing resources can be exacerbated when the data centeris unable to quickly isolate and correct the cause of a connectionfailure. Additionally, scheduled maintenance and replacement of a Tier-0device may require that the servers connected to that Tier-0 device loseconnectivity while the Tier-0 device is being replaced or serviced. Manyservice agreements require that customers are given advance notice, thusprecluding the possibility of scheduling maintenance as needed by theservice provider without providing notice, or precluding the possibilityof replacing the Tier-0 device on the spot.

It is with respect to these considerations and others that thedisclosure made herein is presented.

SUMMARY

The disclosed embodiments describe technologies for providing analternate network path for servers connected to a Tier-0 device to helpprevent servers from becoming isolated from the main data plane networkwhen the Tier-0 device or connections to the Tier-0 device are lost. Thetechnologies may be implemented in conjunction with servers and otherdevices that require network resiliency.

Cloud service providers typically run workloads that are missioncritical and are highly sensitive to server failures. In some cases, thecost of lost business may exceed the deployment costs when failuresoccur.

It may be possible to use a combination of Network Interface Card (NIC)teaming and multi-chassis link aggregation to provide a first and secondinterface into the data plane network to guard against a single cablefailure or single Tier-0 device failure. These methods may be suited foroperating systems and applications that are configured to recognize theadditional network resources and detect and respond to a failure in oneof the network connections. However, NIC teaming and multi-chassis linkaggregation may be difficult to implement for cloud-scale hosted virtualmachine services because some operating systems and softwareapplications may not be designed to recognize two network interfaces.Additionally, there may be issues with a possible loss of trafficforwarding capability that may be difficult to prevent and debug.

In some cases, in order to provide network resiliency down to the serverlevel, it may be possible to connect each server to two different Tier-0network elements through diversely routed facilities (e.g., optical orcopper cables). However, providing a second NIC to each server may becostly when using custom NICs with complex acceleration logic.

It may also be possible to duplicate the Tier-0 devices using a switchor multiplexer. However, this would require changes to the Tier-0 deviceinterfaces that may undermine compatibility between devices in a datacenter, as well as require the expense of custom switches ormultiplexers that can greatly increase the cost of deploying such asolution across a data center or across multiple data centers.

The disclosed embodiments describe a way to respond to a failure of aTier-0 device or failed network facilities (e.g., cabling) and enabledata traffic flows through the network so that normal networkavailability may be quickly restored while avoiding significant impactsto the customers. The disclosed embodiments also enable the replacementof components during maintenance and allow data traffic flows tocontinue through the network without loss of network availability. Anynetwork outage that persists for more than one second, for example, maybe deemed over the threshold for what is considered an outage.Additionally, some software applications may be sensitive to networkimpairments that persist for one second or more. The disclosed methodsincorporate techniques that can quickly react to failures to rearrangetraffic flows into the hardware-provided alternate network path betweena server and the Tier-0 network elements.

In an embodiment, a smart cable is disclosed that enables theimplementation of dual redundant Tier-0 devices (e.g., ToRs) whileminimizing the cost of selecting and switching between the dual ToRs. Inone embodiment, an active direct attach copper/direct attach cable (DAC)assembly is used to switch between dual ToRs. One end of the cable mayuse a standard NIC connector (e.g., QSFP28 or other) while the other endof the cable may have an active multiplexer/switch that selects theactive (primary) ToR.

In an embodiment, both ToRs may receive the same data from eachconnected server and only one (active or primary) ToR will transmit datatowards the connected servers. In one embodiment, the smart cable mayinclude a Serializer/Deserializer (SERDES) based multiplexer thatselects one of the dual ToRs as active or primary.

In some implementations, a command to switch between dual ToRs may besent from the ToRs. The command may be sent through an Inter-IntegratedCircuit (I2C) bus or by writing to a register value. Other means may beimplemented in various embodiments. For example, the NIC may control themultiplexer in the smart cable. In one embodiment, a BidirectionalForwarding Detection (BFD) protocol may be implemented to detect a faultand determine whether to switch ToRs. The detection of a fault of theactive or primary ToR can be used to switch to the secondary ToR.

By implementing a smart cable to switch between dual redundant ToRs, theneed for a NIC to have two uplinks may be eliminated. According to thepresent disclosure, the NIC would not need to be modified to recognizetwo ToRs, since the NIC on the server is only receiving data from oneToR. When the primary ToR is switched to the secondary ToR, the switchcan be performed quickly and thus avoid a potential outage.

The implementation of dual ToRs with a smart cable can allow switchingbetween ToRs during software upgrades or on any detected failure. Any orall of the servers in the rack may be protected in the same way.Furthermore, interfaces to the ToR do not need to be changed toimplement the described embodiments, therefore saving significant coststhat would other be required if the ToR needed to be modified.

Among many other benefits, the techniques shown herein improveefficiencies with respect to a wide range of computing resources. Forinstance, data centers may avoid or reduce the number of serverconnectivity issues and the resulting loss of data and services.Additionally, scheduled maintenance and replacement of Tier-0 devicesmay be performed without losing connectivity to the servers connected tothat Tier-0 device. Additionally, data centers can perform such serviceswithout the requirement to provide advance notice to customers, thusproviding greater efficiencies in scheduling maintenance or performingmaintenance in quick response to incidents. Other technical effects,other than those mentioned herein, can also be realized fromimplementations of the technologies disclosed herein.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intendedthat this Summary be used to limit the scope of the claimed subjectmatter. Furthermore, the claimed subject matter is not limited toimplementations that solve any or all disadvantages noted in any part ofthis disclosure.

DRAWINGS

The Detailed Description is described with reference to the accompanyingfigures. In the description detailed herein, references are made to theaccompanying drawings that form a part hereof, and that show, by way ofillustration, specific embodiments or examples. The drawings herein arenot drawn to scale. Like numerals represent like elements throughout theseveral figures.

FIG. 1 is a diagram illustrating a data center for providing computingresources in accordance with the present disclosure;

FIG. 2A is an example resilient network topology in accordance with thepresent disclosure;

FIG. 2B is an example resilient network topology in accordance with thepresent disclosure;

FIG. 3A is an example resilient network topology in accordance with thepresent disclosure;

FIG. 3B is an example resilient network topology in accordance with thepresent disclosure;

FIG. 4 is a flowchart depicting an example procedure in accordance withthe present disclosure;

FIG. 5 is a flowchart depicting an example procedure in accordance withthe present disclosure;

FIG. 6 is a flowchart depicting an example procedure in accordance withthe present disclosure;

FIG. 7 is an example computing device in accordance with the presentdisclosure.

DETAILED DESCRIPTION

The disclosed embodiments describe technologies for providing analternate network path to prevent servers from becoming isolated fromthe main data plane network. In an embodiment, a multiplexer or switchdevice may be integrated at one end of a DAC cable, which may bereferred to herein as a smart cable. The multiplexer or switch devicemay be referred to herein as a switch device. This smart cable mayprovide a cross-point switch capability that can switch traffic at OSImodel Layer 1 (physical layer) between dual redundant Tier-0 devices,which in many examples herein can be a Top of Rack (ToR) device, and aserver of a group of servers may be connected to the smart cable.

In one embodiment, the smart cable may interconnect one server of a rackto two ToRs. Multiple smart cables may be used to connect each server ofthe rack to the two ToRs. The smart cable may be interchangeable. Thesmart cable may be configured to switch data traffic from a first ToR toa second ToR, for example if the first ToR has failed or needs to beserviced. In an embodiment, circuitry to implement this switchingcapability may be embedded in a Quad Small Form-factor Pluggable(QSFP)-based DAC cable. In various embodiments, various types of DACcables and connector types may be used to implement the describedtechniques, including cables and connectors capable of speeds rangingfrom 10 Gbps-800 Gbps. The circuitry may be configured to switch datatraffic from one ToR device to the other ToR device.

Additionally, an out-of-band control plane signal or indication may beprovided. In one embodiment, Clock and/or I2C lines of the DAC cable maybe used to indicate which of the dual ToRs should be primary. In oneembodiment, the dual ToRs may be configured to determine which ToRshould be primary and generate the indication using the control signal.Because the smart cable is an active DAC, in some embodiments, power maybe provided by the ToRs. Other methods of communication from the ToR tothe NIC may be implemented, such as using ON/OFF of the link which mayavoid extending the I2C control from the ToR to the switch device.Additionally, if BFD is used between the ToR and the NIC, the NIC itselfmay cause the switch device to switch between ToRs based on theabsence/presence of BFD responses. One or more software implementationsmay be used to either allow the ToR to signal the switch device or theNIC to control when to switch between ToRs. Various signaling techniquesmay thus be implemented to accommodate such operations or conditions,such as an entire ToR going offline.

In some embodiments, the switch device in the cable itself may beconfigured to make the decision to switch between ToRs. In oneembodiment, the switch device may be accompanied by a microcontrollerunit (MCU) device configured to control various aspects ofauto-negotiation and link training. This MCU device may execute codethat is modifiable. Interactions with the switch device state can beused to determine if a link is up/down or in some other intermediatestate. The presence of a fully trained link may be interpreted as thelink being in an UP state and hence may be used to allow the switchdevice to determine which link will be used based on a flexible andprogrammable model. In some embodiments, the switch device with MCU maybe configured to work in conjunction with other mechanisms deployed bythe NIC or ToR.

The disclosed techniques allow for maintaining network connectivity toone or more servers of a rack if its connection to the ToR fails or ifone of the ToRs is serviced/replaced. Protection against equipmentfailure of the ToR may be enabled by providing the redundant ToR andproviding a method of quickly switching data traffic flows through thealternate path.

Furthermore, by using such a cable, from the server perspective, only asingle link to a single ToR is observed because data is sent to themultiplexer which switches between the ToRs. No changes are thusrequired on the server side. A switchover of active/standby ToRs at oneserver may not affect other servers in the rack or row. In someembodiments, a rack may support a mix of servers selecting between theprimary and the secondary ToR.

It should be understood that the methods described herein can beextended to two or more ToRs. The described examples are illustratedusing two ToRs for simplicity; however, the techniques may be extendedto two or more ToRs in various implementations. It should also beunderstood that the described techniques may be used using connectorsother than QSFP connectors. For example, the described techniques may beimplemented with DAC connectors configured to operate from 10 Gbps-800Gbps.

FIG. 1 illustrates an example computing environment in which theembodiments described herein may be implemented. FIG. 1 illustrates adata center 100 that configured to provide computing resources to users100 a, 100 b, or 100 c (which may be referred herein singularly as “auser 100” or in the plural as “the users 100”) via user computers 102 a,102 b, and 102 c (which may be referred herein singularly as “a computer102” or in the plural as “the computers 102”) via a communicationsnetwork 130. The computing resources provided by the data center 100 mayinclude various types of resources, such as computing resources, datastorage resources, data communication resources, and the like. Each typeof computing resource may be general-purpose or may be available in anumber of specific configurations. For example, computing resources maybe available as virtual machines. The virtual machines may be configuredto execute applications, including Web servers, application servers,media servers, database servers, and the like. Data storage resourcesmay include file storage devices, block storage devices, and the like.Each type or configuration of computing resource may be available indifferent configurations, such as the number of processors, and size ofmemory and/or storage capacity. The resources may in some embodiments beoffered to clients in units referred to as instances, such as virtualmachine instances or storage instances. A virtual computing instance maybe referred to as a virtual machine and may, for example, comprise oneor more servers with a specified computational capacity (which may bespecified by indicating the type and number of CPUs, the main memorysize and so on) and a specified software stack (e.g., a particularversion of an operating system, which may in turn run on top of ahypervisor).

Data center 100 may include servers 116 a, 116 b, and 116 c (which maybe referred to herein singularly as “a server 116” or in the plural as“the servers 116”) that provide computing resources available as virtualmachines 118 a and 118 b (which may be referred to herein singularly as“a virtual machine 118” or in the plural as “the virtual machines 118”).The virtual machines 118 may be configured to execute applications suchas Web servers, application servers, media servers, database servers,and the like. Other resources that may be provided include data storageresources (not shown on FIG. 1) and may include file storage devices,block storage devices, and the like. Servers 116 may also executefunctions that manage and control allocation of resources in the datacenter, such as a controller 115. Controller 115 may be a fabriccontroller or another type of program configured to manage theallocation of virtual machines on servers 116.

Referring to FIG. 1, communications network 130 may, for example, be apublicly accessible network of linked networks and may be operated byvarious entities, such as the Internet. In other embodiments,communications network 130 may be a private network, such as a corporatenetwork that is wholly or partially inaccessible to the public.

Communications network 130 may provide access to computers 102.Computers 102 may be computers utilized by users 100. Computer 102 a,102 b or 102 c may be a server, a desktop or laptop personal computer, atablet computer, a smartphone, a set-top box, or any other computingdevice capable of accessing data center 100. User computer 102 a or 102b may connect directly to the Internet (e.g., via a cable modem). Usercomputer 102 c may be internal to the data center 100 and may connectdirectly to the resources in the data center 100 via internal networks.Although only three user computers 102 a, 102 b, and 102 c are depicted,it should be appreciated that there may be multiple user computers.

Computers 102 may also be utilized to configure aspects of the computingresources provided by data center 100. For example, data center 100 mayprovide a Web interface through which aspects of its operation may beconfigured through the use of a Web browser application programexecuting on user computer 102. Alternatively, a stand-alone applicationprogram executing on user computer 102 may be used to access anapplication programming interface (API) exposed by data center 100 forperforming the configuration operations.

Servers 116 may be configured to provide the computing resourcesdescribed above. One or more of the servers 116 may be configured toexecute a manager 110 a or 110 b (which may be referred hereinsingularly as “a manager 110” or in the plural as “the managers 110”)configured to execute the virtual machines. The managers 110 may be avirtual machine monitor (VMM), fabric controller, or another type ofprogram configured to enable the execution of virtual machines 118 onservers 116, for example.

It should be appreciated that although the embodiments disclosed aboveare discussed in the context of virtual machines, other types ofimplementations can be utilized with the concepts and technologiesdisclosed herein. For example, the embodiments disclosed herein mightalso be utilized with computing systems that do not utilize virtualmachines.

In the example data center 100 shown in FIG. 1, a router 111 may beutilized to interconnect the servers 116 a and 116 b. Router 111 mayalso be connected to gateway 140, which is connected to communicationsnetwork 130. Router 111 may manage communications within networks indata center 100, for example, by forwarding packets or other datacommunications as appropriate based on characteristics of suchcommunications (e.g., header information including source and/ordestination addresses, protocol identifiers, etc.) and/or thecharacteristics of the private network (e.g., routes based on networktopology, etc.). It will be appreciated that, for the sake ofsimplicity, various aspects of the computing systems and other devicesof this example are illustrated without showing certain conventionaldetails. Additional computing systems and other devices may beinterconnected in other embodiments and may be interconnected indifferent ways.

It should be appreciated that the network topology illustrated in FIG. 1has been greatly simplified and that many more networks and networkingdevices may be utilized to interconnect the various computing systemsdisclosed herein. These network topologies and devices should beapparent to those skilled in the art.

It should also be appreciated that data center 100 described in FIG. 1is merely illustrative and that other implementations might be utilized.Additionally, it should be appreciated that the functionality disclosedherein might be implemented in software, hardware or a combination ofsoftware and hardware. Other implementations should be apparent to thoseskilled in the art. It should also be appreciated that a server,gateway, or other computing device may comprise any combination ofhardware or software that can interact and perform the described typesof functionality, including without limitation desktop or othercomputers, database servers, network storage devices and other networkdevices, tablets, and various other devices that include appropriatecommunication capabilities. In addition, the functionality provided bythe illustrated modules may in some embodiments be combined in fewermodules or distributed in additional modules. Similarly, in someembodiments the functionality of some of the illustrated modules may notbe provided and/or other additional functionality may be available.

FIG. 2A illustrates an example topology of a network implementing asmart cable. In one embodiment, a smart cable may have 4 TX, 4 RX, andone or more control lines. For example, the control lines may include aClock and a I2C line. FIG. 2A illustrates a resilient network topologywith switching implemented at the QSFP cable. Illustrated are two Tier-0ToR network elements 260 and 270. Switching chip 230 may be implementedin the smart cable that implements Layer-1 switching within the cable.One of dual ToRs 260 and 270 are connected to server 210 based on theswitching.

In some embodiments, control and status signaling may be implemented toindicate an active communication path corresponding to one of two ToRs.In some embodiments, the control and status signaling may be implementedas an in-band signal. In other embodiments, out of band control andstatus signaling may be implemented using existing conductors in thesmart cable.

In one embodiment, the control and status signal can be a 2-levelactive/standby signal or a serial bus with multi-master capability.Changes in the active/standby state can be driven by either ToR. In someembodiments, the secondary or slave ToR may initiate a switch if theprimary ToR fails to generate heartbeat messages for a predeterminedthreshold.

FIG. 2B illustrates another example topology of a network implementing asmart cable. In an embodiment, a smart cable may have 4 TX and 4 RX, butdoes not provide control lines from the ToRs. In this example, the NIC215 may determine whether to control the switch device 230 to switchbetween ToRs 260 or 270 based on the presence or absence of a BFDheartbeat for a predetermined threshold, which indicates whether thecurrently active ToR is communicating properly.

FIG. 3A and FIG. 3B illustrate an example smart cable 340 that may beimplemented as disclosed herein. In this example, smart cable 340 may beconnected to diverse network interfaces at TOR1 310 and TOR2 320. Asshown in FIG. 3A, ToR1 310 and ToR2 320 are communicatively coupled.Smart cable 340 may be connected to the server-side interface at server1 350. A switch 340 is controllable via a control signal to select oneof the network interfaces. In an embodiment, a process may be executedto determine which ToR is the master and which ToR will be the slave.Once the master and slave roles have been determined, one or both of theToRs may send an indication to the smart cable identifying which ToRshould be used as the primary for data traffic to the servers. Thefunctionality for determining the primary ToR may be implemented insoftware in the ToR. The servers and their associated network interfacedevices such as the NIC thus do not require new functionality.

In an embodiment, the same data may be transmitted to both ToRs from thenetwork. The switch mechanism 340 in the smart cable 330 may beconfigured to switch between the ToRs. In some embodiments, the switchmechanism 340 may determine which ToR is primary if an error or sometype of inconsistency is detected, for example if both ToRs indicatethat they are primary. When the ToR is switched, the Border GatewayProtocol (BGP) status of the TORs may be updated.

In some embodiments, the switching mechanism may be implemented as adevice such as a chip that includes multiplexing capability. In oneexample, a SERDES based multiplexer may be implemented that isconfigured to select a primary ToR. In one example, the multiplexer canbe configured to provide 8 individual TX/RX lanes on a first side, and 8individual TX/RX lanes on a second side. The multiplexer may beconfigured to support applications for 400G, 200G, 100G, 50G, 40G, 25G,and 10G IEEE Ethernet standards, as well as other speeds.

In some embodiments, the switch 340 may be situated in proximity to thenetwork interface to the server. This example is illustrated in FIG. 3A.

Referring to FIG. 3B, in some embodiments, the switch mechanism 340 maybe situated closer to the junction where the cable splits to ToR1 310and ToR2 320. This may be desirable to reduce the cost of the cable, andwhere the installation environment provides sufficient physical space toaccommodate the switch portion of the cable when a number of cables maybe grouped together in the vicinity of the ToRs.

In another embodiment, the switch mechanism may be situated on a patchpanel that is configured to receive connections from the servers androute the connections to one of two redundant ToRs.

In some embodiments, the ToR connection for a single server can bemoved, rather than for all servers in the rack. In this scenario, eachToR may be active with respect to at least one server in the rack.

In some embodiments, the switching mechanism in the smart cable mayreceive a signal to switch ToRs from the server side rather than fromthe TORs. For example, the NIC may control the multiplexer in the smartcable. In one embodiment, a Bidirectional Forwarding Detection (BFD)protocol may be implemented to detect a fault and determine whether toswitch ToRs. The detection of a fault of the active or primary ToR canbe used to switch to the secondary ToR.

Turning now to FIG. 4, illustrated is an example operational procedurefor implementing a fault resilient mechanism in accordance with thepresent disclosure. The operational procedure may be implemented in asystem comprising plurality of servers and at least two top-of-rackswitches. Referring to FIG. 4, operation 401 illustrates on each of thetwo top-of-rack switches, duplicating data signals at network interfacesof the two top-of-rack switches. In an embodiment, each of the networkinterfaces are communicatively coupled to a data cable having aswitching mechanism configured to select one of the network interfacesof the two top-of-rack switches. In an embodiment, the data cable iscommunicatively coupled to one of the plurality of servers.

Operation 401 may be followed by operation 403. Operation 403illustrates activating a control signal to indicate an activecommunication path from a first of the two top-of-rack switches to thecommunicatively coupled server. In an embodiment, the activecommunication path corresponds to a first of the two network interfaces.

Operation 403 may be followed by operation 405. Operation 405illustrates in response to detecting a failure of the activecommunication path, modifying the control signal to indicate a switch toa second of the two network interfaces corresponding to a second of thetwo top-of-rack switches.

Operation 405 may be followed by operation 407. Operation 407illustrates in response to the modified control signal, switching, bythe switching mechanism of the data cable, the active communication pathto the second of the two network interfaces.

In an embodiment, the data cable is a Direct Attach Cable (DAC). In anembodiment, the data cable is a QSFP28 cable. In an embodiment, thecontrol signal is an out-of-band control plane signal implemented usinga conductor on the DAC. In some embodiments, the out-of-band controlplane signal is one of a 2-level signal or a serial bus. In anembodiment, the control signal is carried on a I2C bus of the data cable

In an embodiment, detecting the failure of the active communication pathand modifying the control signal is performed by at least one of the twotop-of-rack switches. In some embodiments, detecting the failure of theactive communication path and modifying the control signal is performedby a network interface card (NIC) of the communicatively coupled server.

In an embodiment, the failure is detected when a network element of theactive communication path fails to generate heartbeat messages for apredetermined duration.

Turning now to FIG. 4, illustrated is an example operational procedurefor implementing a fault resilient mechanism in accordance with thepresent disclosure. It should be understood that the operations of themethods disclosed herein are not presented in any particular order andthat performance of some or all of the operations in an alternativeorder(s) is possible and is contemplated. The operations have beenpresented in the demonstrated order for ease of description andillustration. Operations may be added, omitted, and/or performedsimultaneously, without departing from the scope of the appended claims.

It also should be understood that the illustrated methods can end at anytime and need not be performed in their entireties. Some or alloperations of the methods, and/or substantially equivalent operations,can be performed by execution of computer-readable instructions includedon a computer-storage media, as defined below. The term“computer-readable instructions,” and variants thereof, as used in thedescription and claims, is used expansively herein to include routines,applications, application modules, program modules, programs,components, data structures, algorithms, and the like. Computer-readableinstructions can be implemented on various system configurations,including single-processor or multiprocessor systems, minicomputers,mainframe computers, personal computers, hand-held computing devices,microprocessor-based, programmable consumer electronics, combinationsthereof, and the like.

Thus, it should be appreciated that the logical operations describedherein are implemented (1) as a sequence of computer implemented acts orprogram modules running on a computing system and/or (2) asinterconnected machine logic circuits or circuit modules within thecomputing system. The implementation is a matter of choice dependent onthe performance and other requirements of the computing system.Accordingly, the logical operations described herein are referred tovariously as states, operations, structural devices, acts, or modules.These operations, structural devices, acts, and modules may beimplemented in software, in firmware, in special purpose digital logic,and any combination thereof.

For example, the operations of the routine 700 are described herein asbeing implemented, at least in part, by modules running the featuresdisclosed herein and can be a dynamically linked library (DLL), astatically linked library, functionality produced by an applicationprograming interface (API), a compiled program, an interpreted program,a script or any other executable set of instructions. Data can be storedin a data structure in one or more memory components. Data can beretrieved from the data structure by addressing links or references tothe data structure.

Although the following illustration refers to the components of thefigures, it can be appreciated that the operations of the routine 400may be also implemented in many other ways. For example, the routine 400may be implemented, at least in part, by a processor of another remotecomputer or a local circuit. In addition, one or more of the operationsof the routine 400 may alternatively or additionally be implemented, atleast in part, by a chipset working alone or in conjunction with othersoftware modules. In the example described below, one or more modules ofa computing system can receive and/or process the data disclosed herein.Any service, circuit or application suitable for providing thetechniques disclosed herein can be used in operations described herein.

The operational procedure may be implemented in a system comprising aplurality of servers and at least two top-of-rack switches. The serversmay be communicatively coupled to network interfaces of the top-of-rackswitches using a plurality of data cables. The data cables may eachcomprise a switch device configured to switch communication pathsbetween the coupled top-of-rack switches. Each of the data cables maycommunicatively couple each of the top-of-rack switches to one of theplurality of servers so that each of the servers have a switchablecommunications path to each of the top-of-rack switches. Referring toFIG. 4, operation 401 illustrates duplicating data signals at thenetwork interfaces of the top-of-rack switches.

Operation 401 may be followed by operation 403. Operation 403illustrates determining validity of a communication path from a first ofthe top-of-rack switches to a first of the communicatively coupledservers. In an embodiment, the communication path may correspond to afirst of the data cables connecting a first of the network interfaces tothe first communicatively coupled server.

Operation 403 may be followed by operation 405. Operation 405illustrates activating a control signal to indicate the communicationpath from the first of the top-of-rack switches to the firstcommunicatively coupled server when the communication path is determinedto be valid.

Operation 405 may be followed by operation 407. Operation 407illustrates in response to the activated control signal, causing theswitch device of the first data cable to connect the first networkinterface to the first communicatively coupled server.

Operation 407 may be followed by operation 409. Operation 409illustrates in response to detecting a failure of the communicationpath, modifying the control signal to indicate a switch to a second ofthe network interfaces corresponding to a second of the two top-of-rackswitches.

Operation 409 may be followed by operation 411. Operation 411illustrates in response to the modified control signal, causing theswitch device of the first data cable to connect the second networkinterface to the first communicatively coupled server.

In an embodiment, the data cable is a Direct Attach Cable (DAC). In anembodiment, the data cable is a QSFP28 cable. In an embodiment, thecontrol signal is an out-of-band control plane signal implemented usinga conductor on the DAC. In some embodiments, the out-of-band controlplane signal is one of a 2-level signal or a serial bus. In anembodiment, the control signal is carried on a I2C bus of the data cableor a serial bus.

In an embodiment, detecting the failure of the active communication pathand modifying the control signal is performed by at least one of the twotop-of-rack switches. In some embodiments, detecting the failure of theactive communication path and modifying the control signal is performedby a network interface card (NIC) of the communicatively coupled server.

In an embodiment, the failure is detected when a network element of theactive communication path fails to generate heartbeat messages for apredetermined duration.

Turning now to FIG. 5, illustrated is an example operational procedurefor implementing a fault resilient mechanism in accordance with thepresent disclosure. The operational procedure may be implemented in asystem comprising plurality of servers and at least two top-of-rackswitches. The plurality of data cables may each have a switching device.The servers may be communicatively coupled to network interfaces of thetop-of-rack switches using the plurality of data cables. The switchdevices may be configured to switch communication paths between thetop-of-rack switches. Each of the data cables may be communicativelycouple the top-of-rack switches to one of the plurality of servers.Referring to FIG. 5, operation 501 illustrates indicating an activecommunication path from a first of the top-of-rack switches to theplurality of servers via the plurality of data cables.

Operation 501 may be followed by operation 503. Operation 503illustrates based on the indicated first communication path, enable, bythe switch devices on the plurality of data cables, the firstcommunication path between the first of the top-of-rack switches and theplurality of servers.

Operation 503 may be followed by operation 505. Operation 505illustrates in response to detecting a failure of the firstcommunication path, indicate a second communication path from a secondof the top-of-rack switches to the plurality of servers via theplurality of data cables.

Operation 505 may be followed by operation 507. Operation 507illustrates based on the indicated second communication path, enable, bythe switch devices on the plurality of data cables, the secondcommunication path between the second of the top-of-rack switches andthe plurality of servers.

Turning now to FIG. 6, illustrated is an example operational procedurefor implementing a fault resilient mechanism in accordance with thepresent disclosure. The operational procedure may be implemented in asystem comprising plurality of servers and at least two top-of-rackswitches. Referring to FIG. 6, operation 601 illustrates transmittingand receiving data between the server and a first of the top-of-rackswitches.

Operation 601 may be followed by operation 603. Operation 603illustrates transmitting and receiving data between the server and asecond of the top-of-rack switches.

Operation 603 may be followed by operation 605. Operation 605illustrates receiving a control signal to indicate a valid communicationpath from one of the top-of-rack switches to the server.

Operation 605 may be followed by operation 607. Operation 607illustrates selecting, based on the control signal, either the first setof conductors or the second set of conductors for transmitting andreceiving data between the server and the top-of-rack switches.

In an embodiment, the data cable includes a third set of conductors forreceiving a control signal to indicate an active set of conductors fortransmitting and receiving data between the server and one of thetop-of-rack switches.

In an embodiment, a top-of-rack end of the data cable has a firstconnector for the first set of conductors and a second connector for thesecond set of conductors.

In an embodiment, the data cable is a Direct Attach Cable (DAC). In anembodiment, the data cable is a QSFP28 cable.

In an embodiment, the data cable is configured to select the first setof conductors or the second set of conductors for transmitting andreceiving data between the server and the top-of-rack switches when thecontrol signal is not received.

In an embodiment, the switching circuit is situated proximate to ajunction for the first and second sets of conductors. In anotherembodiment, the switching circuit is situated proximate to a server sideof the data cable.

The various aspects of the disclosure are described herein with regardto certain examples and embodiments, which are intended to illustratebut not to limit the disclosure. It should be appreciated that thesubject matter presented herein may be implemented as a computerprocess, a computer-controlled apparatus, a computing system, an articleof manufacture, such as a computer-readable storage medium, or acomponent including hardware logic for implementing functions, such as afield-programmable gate array (FPGA) device, a massively parallelprocessor array (MPPA) device, a graphics processing unit (GPU), anapplication-specific integrated circuit (ASIC), a multiprocessorSystem-on-Chip (MPSoC), etc. A component may also encompass other waysof leveraging a device to perform a function, such as, for example, a) acase in which at least some tasks are implemented in hard ASIC logic orthe like; b) a case in which at least some tasks are implemented in soft(configurable) FPGA logic or the like; c) a case in which at least sometasks run as software on FPGA software processor overlays or the like;d) a case in which at least some tasks run as software on hard ASICprocessors or the like, etc., or any combination thereof. A componentmay represent a homogeneous collection of hardware acceleration devices,such as, for example, FPGA devices. On the other hand, a component mayrepresent a heterogeneous collection of different types of hardwareacceleration devices including different types of FPGA devices havingdifferent respective processing capabilities and architectures, amixture of FPGA devices and other types hardware acceleration devices,etc.

Those skilled in the art will also appreciate that the subject matterdescribed herein may be practiced on or in conjunction with othercomputer system configurations beyond those described herein, includingmultiprocessor systems. The embodiments described herein may also bepracticed in distributed computing environments, where tasks areperformed by remote processing devices that are linked through acommunications network. In a distributed computing environment, programmodules may be located in both local and remote memory storage devices.

Networks established by or on behalf of a user to provide one or moreservices (such as various types of cloud-based computing or storage)accessible via the Internet and/or other networks to a distributed setof clients may be referred to as a service provider. Such a network mayinclude one or more data centers such as data center 100 illustrated inFIG. 1, which are configured to host physical and/or virtualizedcomputer servers, storage devices, networking equipment and the like,that may be used to implement and distribute the infrastructure andservices offered by the service provider.

In some embodiments, a server that implements a portion or all of one ormore of the technologies described herein, including the techniques toimplement the capturing of network traffic may include a general-purposecomputer system that includes or is configured to access one or morecomputer-accessible media. FIG. 8 illustrates such a general-purposecomputing device 800. In the illustrated embodiment, computing device800 includes one or more processors 810 a, 810 b, and/or 810 n (whichmay be referred herein singularly as “a processor 810” or in the pluralas “the processors 810”) coupled to a system memory 820 via aninput/output (I/O) interface 830. Computing device 800 further includesa network interface 840 coupled to I/O interface 830.

In various embodiments, computing device 800 may be a uniprocessorsystem including one processor 810 or a multiprocessor system includingseveral processors 810 (e.g., two, four, eight, or another suitablenumber). Processors 810 may be any suitable processors capable ofexecuting instructions. For example, in various embodiments, processors810 may be general-purpose or embedded processors implementing any of avariety of instruction set architectures (ISAs), such as the x86,PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. Inmultiprocessor systems, each of processors 810 may commonly, but notnecessarily, implement the same ISA.

System memory 820 may be configured to store instructions and dataaccessible by processor(s) 810. In various embodiments, system memory820 may be implemented using any suitable memory technology, such asstatic random access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques and datadescribed above, are shown stored within system memory 820 as code 825and data 826.

In one embodiment, I/O interface 880 may be configured to coordinate I/Otraffic between the processor 810, system memory 820, and any peripheraldevices in the device, including network interface 840 or otherperipheral interfaces. In some embodiments, I/O interface 880 mayperform any necessary protocol, timing, or other data transformations toconvert data signals from one component (e.g., system memory 820) into aformat suitable for use by another component (e.g., processor 810). Insome embodiments, I/O interface 880 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 880 may be split into two or more separatecomponents. Also, in some embodiments some or all of the functionalityof I/O interface 880, such as an interface to system memory 820, may beincorporated directly into processor 810.

Network interface 840 may be configured to allow data to be exchangedbetween computing device 800 and other device or devices 860 attached toa network or network(s) 850, such as other computer systems or devicesas illustrated in FIGS. 1 through 4, for example. In variousembodiments, network interface 840 may support communication via anysuitable wired or wireless general data networks, such as types ofEthernet networks, for example. Additionally, network interface 840 maysupport communication via telecommunications/telephony networks such asanalog voice networks or digital fiber communications networks, viastorage area networks such as Fibre Channel SANs or via any othersuitable type of network and/or protocol.

In some embodiments, system memory 820 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above for FIGS. 1-7 for implementing embodiments ofthe corresponding methods and apparatus. However, in other embodiments,program instructions and/or data may be received, sent or stored upondifferent types of computer-accessible media. A computer-accessiblemedium may include non-transitory storage media or memory media, such asmagnetic or optical media, e.g., disk or DVD/CD coupled to computingdevice 800 via I/O interface 880. A non-transitory computer-accessiblestorage medium may also include any volatile or non-volatile media, suchas RAM (e.g. SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that maybe included in some embodiments of computing device 800 as system memory820 or another type of memory. Further, a computer-accessible medium mayinclude transmission media or signals such as electrical,electromagnetic or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link, such as may be implemented vianetwork interface 840. Portions or all of multiple computing devices,such as those illustrated in FIG. 8, may be used to implement thedescribed functionality in various embodiments; for example, softwarecomponents running on a variety of different devices and servers maycollaborate to provide the functionality. In some embodiments, portionsof the described functionality may be implemented using storage devices,network devices, or special-purpose computer systems, in addition to orinstead of being implemented using general-purpose computer systems. Theterm “computing device,” as used herein, refers to at least all thesetypes of devices and is not limited to these types of devices.

Various storage devices and their associated computer-readable mediaprovide non-volatile storage for the computing devices described herein.Computer-readable media as discussed herein may refer to a mass storagedevice, such as a solid-state drive, a hard disk or CD-ROM drive.However, it should be appreciated by those skilled in the art thatcomputer-readable media can be any available computer storage media thatcan be accessed by a computing device.

By way of example, and not limitation, computer storage media mayinclude volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. For example, computer media includes, but is not limited to,RAM, ROM, EPROM, EEPROM, flash memory or other solid state memorytechnology, CD-ROM, digital versatile disks (“DVD”), HD-DVD, BLU-RAY, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bythe computing devices discussed herein. For purposes of the claims, thephrase “computer storage medium,” “computer-readable storage medium” andvariations thereof, does not include waves, signals, and/or othertransitory and/or intangible communication media, per se.

Encoding the software modules presented herein also may transform thephysical structure of the computer-readable media presented herein. Thespecific transformation of physical structure may depend on variousfactors, in different implementations of this description. Examples ofsuch factors may include, but are not limited to, the technology used toimplement the computer-readable media, whether the computer-readablemedia is characterized as primary or secondary storage, and the like.For example, if the computer-readable media is implemented assemiconductor-based memory, the software disclosed herein may be encodedon the computer-readable media by transforming the physical state of thesemiconductor memory. For example, the software may transform the stateof transistors, capacitors, or other discrete circuit elementsconstituting the semiconductor memory. The software also may transformthe physical state of such components in order to store data thereupon.

As another example, the computer-readable media disclosed herein may beimplemented using magnetic or optical technology. In suchimplementations, the software presented herein may transform thephysical state of magnetic or optical media, when the software isencoded therein. These transformations may include altering the magneticcharacteristics of particular locations within given magnetic media.These transformations also may include altering the physical features orcharacteristics of particular locations within given optical media, tochange the optical characteristics of those locations. Othertransformations of physical media are possible without departing fromthe scope and spirit of the present description, with the foregoingexamples provided only to facilitate this discussion.

In light of the above, it should be appreciated that many types ofphysical transformations take place in the disclosed computing devicesin order to store and execute the software components and/orfunctionality presented herein. It is also contemplated that thedisclosed computing devices may not include all of the illustratedcomponents shown in FIG. 8, may include other components that are notexplicitly shown in FIG. 8, or may utilize an architecture completelydifferent than that shown in FIG. 8.

Although the various configurations have been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appendedrepresentations is not necessarily limited to the specific features oracts described. Rather, the specific features and acts are disclosed asexample forms of implementing the claimed subject matter.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements, and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements, and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements, and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

While certain example embodiments have been described, these embodimentshave been presented by way of example only, and are not intended tolimit the scope of the inventions disclosed herein. Thus, nothing in theforegoing description is intended to imply that any particular feature,characteristic, step, module, or block is necessary or indispensable.Indeed, the novel methods and systems described herein may be embodiedin a variety of other forms; furthermore, various omissions,substitutions and changes in the form of the methods and systemsdescribed herein may be made without departing from the spirit of theinventions disclosed herein. The accompanying claims and theirequivalents are intended to cover such forms or modifications as wouldfall within the scope and spirit of certain of the inventions disclosedherein.

It should be appreciated any reference to “first,” “second,” etc. itemsand/or abstract concepts within the description is not intended to andshould not be construed to necessarily correspond to any reference of“first,” “second,” etc. elements of the claims. In particular, withinthis Summary and/or the following Detailed Description, items and/orabstract concepts such as, for example, individual computing devicesand/or operational states of the computing cluster may be distinguishedby numerical designations without such designations corresponding to theclaims or even other paragraphs of the Summary and/or DetailedDescription. For example, any designation of a “first operational state”and “second operational state” of the computing cluster within aparagraph of this disclosure is used solely to distinguish two differentoperational states of the computing cluster within that specificparagraph—not any other paragraph and particularly not the claims.

In closing, although the various techniques have been described inlanguage specific to structural features and/or methodological acts, itis to be understood that the subject matter defined in the appendedrepresentations is not necessarily limited to the specific features oracts described. Rather, the specific features and acts are disclosed asexample forms of implementing the claimed subject matter.

1. A method for routing data at a rack comprising a plurality of servers and at least two top-of-rack switches, the servers communicatively coupled to network interfaces of the top-of-rack switches using a plurality of data cables, the data cables each comprising a switch device configured to switch communication paths between the coupled top-of-rack switches, each of the data cables communicatively coupling each of the top-of-rack switches to one of the plurality of servers so that each of the servers have a switchable communications path to each of the top-of-rack switches, the method comprising: duplicating data signals at the network interfaces of the top-of-rack switches; determine validity of a communication path from a first of the top-of-rack switches to a first of the communicatively coupled servers, the communication path corresponding to a first of the data cables connecting a first of the network interfaces to the first communicatively coupled server; activating a control signal to indicate the communication path from the first of the top-of-rack switches to the first communicatively coupled server when the communication path is determined to be valid; in response to the activated control signal, causing the switch device of the first data cable to connect the first network interface to the first communicatively coupled server; in response to detecting a failure of the communication path, modifying the control signal to indicate a switch to a second of the network interfaces corresponding to a second of the two top-of-rack switches; and in response to the modified control signal, causing the switch device of the first data cable to connect the second network interface to the first communicatively coupled server.
 2. The method of claim 1, wherein the data cable is a Direct Attach Cable (DAC).
 3. The method of claim 2, wherein the control signal is an out-of-band control plane signal implemented using a conductor on the DAC.
 4. The method of claim 3, wherein the out-of-band control plane signal is one of a 2-level signal or a serial bus.
 5. The method of claim 1, wherein detecting the failure of the communication path and modifying the control signal is performed by at least one of the top-of-rack switches.
 6. The method of claim 1, wherein detecting the failure of the communication path and modifying the control signal is performed by a network interface card (NIC) of the first communicatively coupled server.
 7. The method of claim 1, wherein the failure is detected when a network element of the communication path fails to generate heartbeat messages for a predetermined duration.
 8. The method of claim 1, wherein the data cable is a QSFP28 cable.
 9. The method of claim 1, wherein the control signal is carried on a I2C bus of the data cable.
 10. A system comprising: a plurality of servers; at least two top-of-rack switches; and a plurality of data cables each having a switching device, the servers communicatively coupled to network interfaces of the top-of-rack switches using the plurality of data cables, the switch devices configured to switch communication paths between the top-of-rack switches, each of the data cables communicatively coupling the top-of-rack switches to one of the plurality of servers; the system configured to: indicate a first communication path from a first of the top-of-rack switches to the plurality of servers via the plurality of data cables; based on the indicated first communication path, enable, by the switch devices on the plurality of data cables, the first communication path between the first of the top-of-rack switches and the plurality of servers; in response to detecting a failure of the first communication path, indicate a second communication path from a second of the top-of-rack switches to the plurality of servers via the plurality of data cables; and based on the indicated second communication path, enable, by the switch devices on the plurality of data cables, the second communication path between the second of the top-of-rack switches and the plurality of servers.
 11. The system of claim 10, wherein the first communication path and second communication path are indicated using a control signal implemented on a conductor on the plurality of data cables.
 12. The system of claim 10, wherein the first communication path and second communication path are determined by the top-of-rack switches.
 13. The system of claim 10, wherein the first communication path and second communication path are determined by a network interface card (NIC) of the plurality of servers.
 14. A data cable for connecting a server to at least two top-of-rack switches, the data cable comprising: a first set of conductors for transmitting and receiving data between the server and a first of the top-of-rack switches; a second set of conductors for transmitting and receiving data between the server and a second of the top-of-rack switches; and a switching circuit configured to select, based on a control signal indicative of a valid communication path from one of the top-of-rack switches to the server, either the first set of conductors or the second set of conductors for transmitting and receiving data between the server and the top-of-rack switches; wherein a top-of-rack end of the data cable has a first connector for the first set of conductors and a second connector for the second set of conductors.
 15. The data cable of claim 14, further comprising a third set of conductors for receiving a control signal to indicate an active set of conductors for transmitting and receiving data between the server and one of the top-of-rack switches.
 16. The data cable of claim 14, wherein the data cable is a Direct Attach Cable (DAC).
 17. The data cable of claim 16, wherein the control signal is an out-of-band control plane signal implemented using a conductor on the DAC.
 18. The data cable of claim 14, wherein the data cable is configured to select the first set of conductors or the second set of conductors for transmitting and receiving data between the server and the top-of-rack switches when the control signal is not received.
 19. The data cable of claim 14, wherein the switching circuit is situated proximate to a junction for the first and second sets of conductors.
 20. The data cable of claim 14, wherein the switching circuit is situated proximate to a server side of the data cable. 